Python recipe for automatic mixed precision #1137
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Deploy preview for pytorch-tutorials-preview ready! Built with commit 3e9815a https://deploy-preview-1137--pytorch-tutorials-preview.netlify.app |
mcarilli
changed the title
|
thanks for submitting this Mike! |
mcarilli
changed the title
|
Article preview here. Might consider removing the links in the page to subsections, in favor of the right nav that contains the same links. |
Not sure what you mean. Do you mean I should replace things like |
|
Thanks @mcarilli, I was suggesting removing that links completely as they already exist on the right.
|
…into amp_tutorial
mcarilli
changed the title
brianjo
added a commit
that referenced
this pull request
Sep 24, 2020
* Fix typo (#1118) In PyTorch tutorial, `torch` should be installed rather than `torchaudio` * Recover the attributes of torch in memory_format_tutorial (#1112) Co-authored-by: Brian Johnson <brianjo@fb.com> * fix bugs for data_loading_tutorial and dcgan_faces_tutorial (#1092) * Update autocast in dispatcher tutorial (#1128) * draft * fixes * dont overrun the line * Corrected model.resnet50() spelling (#1139) Spelling mistake led to errors for beginners. * Fix typo & Minor changes (#1138) Thanks for the fixes @codingbowoo! * Run win_test_worker manually (#1142) Merging to clean up a build issue. * Disable `pytorch_windows_builder_worker` config (#1143) See #1141 * Update index.rst (#1140) Fixed incorrect link. * Update index.rst Fix to broken link. * LSTM's -> LSTMs in equence_models_tutorial.py docs (#1136) Co-authored-by: Brian Johnson <brianjo@fb.com> * Added Ray Tune Hyperparameter Tuning Tutorial (#1066) * Added Ray Tune Hyperparameter Tuning Tutorial * Use nightly ray release * Fix checkpoint API Co-authored-by: Brian Johnson <brianjo@fb.com> * Fix typo in "Introduction to Pytorch" tutorial (in NLP tutorial) (#1145) * Fix typo in "Introduction to Pytorch" tutorial (in Pytorch for NLP tutorials) * Dummy commit, to restart CI * Revert dummy commit, to restart CI * Revert whitespace changes * Install torch not torch vision (#1153) Small update to recipe that instructs users to install `torch` not `torchaudio` * Python recipe for automatic mixed precision (#1137) * fdsa * Tutorial runs * clarify one scaler per convergence run * adjust sizes, dont run illustrative sections * satisfying ocd * MORE * fdsa * details * rephrase * fix formatting * move script to recipes * hopefully moved to recipes * fdsa * add amp_tutorial to toctree * amp_tutorial -> amp_recipe * looks like backtick highlights dont render in card_description * correct path for amp_recipe.html * arch notes and saving/restoring * formatting * fdsa * Clarify autograd-autocast interaction for custom ops * touchups Co-authored-by: Brian Johnson <brianjo@fb.com> * Fix model to be properly exported to ONNX (#1144) Co-authored-by: Brian Johnson <brianjo@fb.com> * Dist rpc merge (#1158) * Create distributed_rpc_profiling.rst * Update recipes_index.rst * Add files via upload * Update recipes_index.rst * Fix typo "asynchronizely" -> "asynchronously" (#1154) * Update dist_overview with additional information. (#1155) Summary: 1) Added DDP + RPC tutorial. 2) Added a pointer to PT Distributed CONTRIBUTING.md. Test Plan: Verified by loading the page locally. Reviewers: sentinel Subscribers: Tasks: Tags: Co-authored-by: pritam <pritam.damania@fb.com> * Add Performance Tuning guide recipe (#1161) * Performance Tuning Guide - initial commit * Minor tweaks * Switched profiling guide thumbnail to pytorch logo * Converted Tuning Guide to 80 chars/line * Split tuning guide into general, GPU-specific and distributed optimizations. * WAR to fix generation of header for 1st section * Minor fixes * Implemented changes suggested during initial review * Changed 'addition assignment' to 'addition' * Removed sentences about 1 CPU core for DataParallel training * Reordering of layers is recommended only for DDP(find_unused_parameters=True) * Fixed formatting * s/constructors/model constructors and s/match/roughly match * Fixed typos * A fix for one line comment when removing runnable code. (#1165) Co-authored-by: v-jizhang <66389669+buck-bot@users.noreply.github.com> Co-authored-by: Nikita Shulga <nshulga@fb.com> Co-authored-by: guyang3532 <62738430+guyang3532@users.noreply.github.com> Co-authored-by: mcarilli <mcarilli@gmail.com> Co-authored-by: Sayantan Das <36279638+ucalyptus@users.noreply.github.com> Co-authored-by: 장보우 Bowoo Jang <lisabowoo@gmail.com> Co-authored-by: Alan deLevie <adelevie@gmail.com> Co-authored-by: krfricke <krfricke@users.noreply.github.com> Co-authored-by: Vijay Viswanathan <vcomeinpeace@gmail.com> Co-authored-by: J. Randall Hunt <randallhunt@gmail.com> Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com> Co-authored-by: Peter Whidden <all.cows.like.to.moo@gmail.com> Co-authored-by: Pritam Damania <9958665+pritamdamania87@users.noreply.github.com> Co-authored-by: pritam <pritam.damania@fb.com> Co-authored-by: Szymon Migacz <1934379+szmigacz@users.noreply.github.com> Co-authored-by: Jinlin Zhang <jinlinzhang@fb.com> Co-authored-by: v-jizhang <66389669+buck-bot@users.noreply.github.com>
rodrigo-techera
pushed a commit
to Experience-Monks/tutorials
that referenced
this pull request
Nov 29, 2021
* fdsa * Tutorial runs * clarify one scaler per convergence run * adjust sizes, dont run illustrative sections * satisfying ocd * MORE * fdsa * details * rephrase * fix formatting * move script to recipes * hopefully moved to recipes * fdsa * add amp_tutorial to toctree * amp_tutorial -> amp_recipe * looks like backtick highlights dont render in card_description * correct path for amp_recipe.html * arch notes and saving/restoring * formatting * fdsa * Clarify autograd-autocast interaction for custom ops * touchups Co-authored-by: Brian Johnson <brianjo@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
^
Still need to tune the model architecture to make mixed precision look really good :-)
After brief slack discussion, the walkthrough is under recipes rather than tutorials. The walkthrough is single-topic, but I wouldn't call it "bite-size," so IMO the final placement (tutorial vs recipe) remains open to bikeshedding based on how it looks in the preview.