Remove legacy torchtext code from translation tutorial #1250
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Deploy preview for pytorch-tutorials-preview ready! Built with commit 087fe98 https://deploy-preview-1250--pytorch-tutorials-preview.netlify.app |
cpuhrsch
reviewed
Nov 20, 2020
| data_.append((de_tensor_, en_tensor_)) | ||
| return data_ | ||
|
|
||
| train_data = data_process(iter(io.open(train_filepaths[0])), |
There was a problem hiding this comment.
I think merging this into the data_process function and then only passing _filepaths cleans this up a bit
cpuhrsch
reviewed
Nov 20, 2020
| de_vocab = build_vocab(train_filepaths[0], de_tokenizer) | ||
| en_vocab = build_vocab(train_filepaths[1], en_tokenizer) | ||
|
|
||
| def data_process(raw_de_iter, raw_en_iter): |
There was a problem hiding this comment.
How long does this function take? How long does the entire prepreocessing step take? Should we tokenizer async while training instead?
There was a problem hiding this comment.
offline discussion: we will have a follow-up PR since this PR doesn't focus on the dataloader.
cpuhrsch
reviewed
Nov 20, 2020
| en_vocab = build_vocab(train_filepaths[1], en_tokenizer) | ||
|
|
||
| def data_process(raw_de_iter, raw_en_iter): | ||
| data_ = [] |
There was a problem hiding this comment.
nit: remove the trailing underscores
cpuhrsch
approved these changes
Nov 20, 2020
zhangguanheng66
changed the title
|
cc @brianjo |
cpuhrsch
reviewed
Nov 30, 2020
| data from a well-known dataset containing sentences in both English and German and use it to | ||
| train a sequence-to-sequence model with attention that can translate German sentences | ||
| into English. | ||
|
|
||
| It is based off of | ||
| `this tutorial <https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb>`__ | ||
| from PyTorch community member `Ben Trevett <https://github.com/bentrevett>`__ | ||
| and was created by `Seth Weidman <https://github.com/SethHWeidman/>`__ with Ben's permission. | ||
| with Ben's permission. We update the tutorials by removing some legecy code. |
There was a problem hiding this comment.
nit: legacy instead of legecy
zhangguanheng66
force-pushed
the
translation_tutorial
branch
2 times, most recently
from
0d504ea to
8c817f1
Compare
zhangguanheng66
force-pushed
the
translation_tutorial
branch
from
a3b3e27 to
2f43983
Compare
rodrigo-techera
pushed a commit
to Experience-Monks/tutorials
that referenced
this pull request
Nov 29, 2021
* checkpoint * checkpoint * minor changes with review's feedback * fix typo * Fix ascii decode error Co-authored-by: Guanheng Zhang <zhangguanheng@devfair0197.h2.fair> Co-authored-by: Brian Johnson <brianjo@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.