Skip to content

Commit fe9ba7d

Browse files
tianyu-lsvekars
authored andcommitted
[Tensor Parallel] remove non-existing code pointer (#3000)
ghstack-source-id: 533d9b7 Pull Request resolved: #2998
1 parent 4220254 commit fe9ba7d

File tree

1 file changed

+1
-3
lines changed

1 file changed

+1
-3
lines changed

intermediate_source/TP_tutorial.rst

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,6 @@ To see how to utilize DeviceMesh to set up multi-dimensional parallelisms, pleas
8383

8484
.. code-block:: python
8585
86-
# run this via torchrun: torchrun --standalone --nproc_per_node=8 ./tp_tutorial.py
87-
8886
from torch.distributed.device_mesh import init_device_mesh
8987
9088
tp_mesh = init_device_mesh("cuda", (8,))
@@ -360,4 +358,4 @@ Conclusion
360358
This tutorial demonstrates how to train a large Transformer-like model across hundreds to thousands of GPUs using Tensor Parallel in combination with Fully Sharded Data Parallel.
361359
It explains how to apply Tensor Parallel to different parts of the model, with **no code changes** to the model itself. Tensor Parallel is a efficient model parallelism technique for large scale training.
362360

363-
To see the complete end to end code example explained in this tutorial, please refer to the `Tensor Parallel examples <https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py>`__ in the pytorch/examples repository.
361+
To see the complete end-to-end code example explained in this tutorial, please refer to the `Tensor Parallel examples <https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py>`__ in the pytorch/examples repository.

0 commit comments

Comments
 (0)