Skip to content

Commit 33d06c3

Browse files
wz337svekars
andauthored
Apply suggestions from code review
Co-authored-by: Svetlana Karslioglu <svekars@meta.com>
1 parent 9a07238 commit 33d06c3

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

recipes_source/distributed_device_mesh.rst

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ Prerequisites:
1111
- `Distributed Communication Package - torch.distributed <https://pytorch.org/docs/stable/distributed.html>`__
1212

1313
.. Setting up the NVIDIA Collective Communication Library (NCCL) communicators for distributed communication during distributed training can pose a significant challenge. For workloads where users need to compose different parallelisms,
14-
.. users would need to manually set up and manage nccl communicators(for example, :class:`ProcessGroup`) for each parallelism solutions. This is fairly complicated and error-proned.
15-
.. :class:`DeviceMesh` can help make this process much easier.
14+
.. users would need to manually set up and manage NCCL communicators (for example, :class:`ProcessGroup`) for each parallelism solutions. This process could be complicated and susceptible to errors.
15+
.. :class:`DeviceMesh` can simplify this process, making it more manageable and less prone to errors.
1616
1717
What is DeviceMesh
1818
------------------
19-
.. :class:`DeviceMesh` is a higher level abstraction that manages :class:`ProcessGroup`. It allows users to easily
19+
.. :class:`DeviceMesh` is a higher level abstraction that manages :class:`ProcessGroup`. It allows users to effortlessly
2020
.. create inter-node and intra-node process groups without worrying about how to set up ranks correctly for different sub process groups.
2121
.. Users can also easily manage the underlying process_groups/devices for multi-dimensional parallelism via :class:`DeviceMesh`.
2222
@@ -28,7 +28,7 @@ What is DeviceMesh
2828
Why DeviceMesh is Useful
2929
------------------------
3030

31-
.. Below is the code snippet for a 2D setup without :class:`DeviceMesh`. First, we need to manually calculate shard group and replicate group. Then, we need to assign the correct shard and
31+
.. The following code snippet illustrates a 2D setup without :class:`DeviceMesh`. First, we need to manually calculate the shard group and replicate group. Then, we need to assign the correct shard and
3232
.. replicate group to each rank.
3333
3434
.. code-block:: python
@@ -57,7 +57,7 @@ current_shard_group = (
5757
shard_groups[0] if rank in shard_rank_lists[0] else shard_groups[1]
5858
)
5959

60-
# Create replicate groups (e.g. (0, 4), (1, 5), (2, 6), (3, 7))
60+
# Create replicate groups (for example, (0, 4), (1, 5), (2, 6), (3, 7))
6161
# and assign the correct replicate group to each rank
6262
current_replicate_group = None
6363
shard_factor = len(shard_rank_lists[0])
@@ -74,7 +74,7 @@ for i in range(num_node_devices // 2):
7474
torchrun --nnodes=1 --nproc_per_node=8 --rdzv_id=100 --rdzv_endpoint=localhost:29400 2d_setup.py
7575

7676

77-
.. With the help of :func:`init_device_mesh`, we can accomplish the above 2D setup in just 2 lines.
77+
.. With the help of :func:`init_device_mesh`, we can accomplish the above 2D setup in just two lines.
7878
7979
8080
.. code-block:: python

0 commit comments

Comments
 (0)