Skip to content

Commit be26e48

Browse files
committed
[Tensor Parallel] remove non-existing code pointer
ghstack-source-id: 533d9b7 Pull Request resolved: #2998
1 parent f1c0b8a commit be26e48

File tree

1 file changed

+1
-3
lines changed

1 file changed

+1
-3
lines changed

intermediate_source/TP_tutorial.rst

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,6 @@ To see how to utilize DeviceMesh to set up multi-dimensional parallelisms, pleas
8383

8484
.. code-block:: python
8585
86-
# run this via torchrun: torchrun --standalone --nproc_per_node=8 ./tp_tutorial.py
87-
8886
from torch.distributed.device_mesh import init_device_mesh
8987
9088
tp_mesh = init_device_mesh("cuda", (8,))
@@ -360,4 +358,4 @@ Conclusion
360358
This tutorial demonstrates how to train a large Transformer-like model across hundreds to thousands of GPUs using Tensor Parallel in combination with Fully Sharded Data Parallel.
361359
It explains how to apply Tensor Parallel to different parts of the model, with **no code changes** to the model itself. Tensor Parallel is a efficient model parallelism technique for large scale training.
362360

363-
To see the complete end to end code example explained in this tutorial, please refer to the `Tensor Parallel examples <https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py>`__ in the pytorch/examples repository.
361+
To see the complete end-to-end code example explained in this tutorial, please refer to the `Tensor Parallel examples <https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py>`__ in the pytorch/examples repository.

0 commit comments

Comments
 (0)