[DeviceMesh] Add device mesh recipe #2718
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/2718
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit b4cb387 with merge base eec8d56 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
|
||
| - `Distributed Communication Package - torch.distributed <https://pytorch.org/docs/stable/distributed.html>`__ | ||
|
|
||
| .. Setting up nccl communicators for distributed communication during distributed training could be challenging. For workloads where users need to compose different parallelisms, |
There was a problem hiding this comment.
I'm not sure why each line has as .. prefix... I think this will render as a directive . If this was meant to be a comment the .. need to be on a separate line.
| How to use DeviceMesh with HSDP | ||
| ------------------------------- | ||
|
|
||
| Hybrid Sharding(HSDP) |
There was a problem hiding this comment.
Would like more info about HSDP. What it is what's the benefits/use cases of using with DeviceMesh, etc.
wz337
force-pushed
the
add_device_mesh_recipe
branch
from
33d06c3 to
fd43ef4
Compare
Co-authored-by: Svetlana Karslioglu <svekars@meta.com>
Co-authored-by: Svetlana Karslioglu <svekars@meta.com>
wz337
force-pushed
the
add_device_mesh_recipe
branch
from
fd43ef4 to
572fdd9
Compare
wz337
force-pushed
the
add_device_mesh_recipe
branch
from
af662e5 to
1102397
Compare
| .. code-block:: python | ||
| torchrun --nnodes=1 --nproc_per_node=8 --rdzv_id=100 --rdzv_endpoint=localhost:29400 2d_setup.py | ||
|
|
||
| Note |
There was a problem hiding this comment.
would this "Note" require some special formatting? i.e. sth like
``note::
And then put the below paragraph after this?
| Then, run the following `torch elastic/torchrun <https://pytorch.org/docs/stable/elastic/quickstart.html>`__ command. | ||
|
|
||
| .. code-block:: python | ||
| torchrun --nnodes=1 --nproc_per_node=8 --rdzv_id=100 --rdzv_endpoint=localhost:29400 hsdp.py |
There was a problem hiding this comment.
ditto: not sure if we want to simplify the run command here
recipes_source/recipes_index.rst
Outdated
| :card_description: Learn how to use DeviceMesh | ||
| :image: ../_static/img/thumbnails/cropped/profiler.png | ||
| :link: ../recipes/distributed_device_mesh.html | ||
| :tags: Distributed-Training |
There was a problem hiding this comment.
shall we put this card to the first of the distributed section? iirc all other receipts are not quite useful anymore (i.e. RPC)
There was a problem hiding this comment.
shall we put this card to the first of the distributed section? iirc all other receipts are not quite useful anymore (i.e. RPC)
We can also add the tutorial on the distributed page as @svekars suggested: https://pytorch.org/tutorials/distributed/home.html
wz337
force-pushed
the
add_device_mesh_recipe
branch
from
5d434d9 to
42592fb
Compare
distributed/home.rst
Outdated
| :link: https://pytorch.org/tutorials/recipes/distributed_device_mesh.html?highlight=devicemesh | ||
| :link-type: url | ||
|
|
||
| In this tutorial you will learn to implement about `DeviceMesh` |
There was a problem hiding this comment.
"implement about"? seems we should say: you will learn about how to use
Co-authored-by: Wanchao <wanchaol@users.noreply.github.com>
* add DeviceMesh recipe --------- Co-authored-by: Svetlana Karslioglu <svekars@meta.com> Co-authored-by: Wanchao <wanchaol@users.noreply.github.com>
* add DeviceMesh recipe --------- Co-authored-by: Svetlana Karslioglu <svekars@meta.com> Co-authored-by: Wanchao <wanchaol@users.noreply.github.com>
Description
Add DeviceMesh tutorial