Attention decoder in seq2seq model different from the one on paper

[ssooffiiaannee]

Following is the attention decoder network used in this tutorial.

But I find it different from the model on this paper, which is the improvement by the attention mechanism.

For example, on the first picture, the embedding vectors go through the attention layer, but in the second one, the embedding vectors (Wich I consider to be the Xs) start first from the bidirectionnal RNN network.
Same for dropout before embedded, I don't see any dropout on the paper.

cc @pytorch/team-text-core @Nayef211

[Alfo5123]

Hi @ssooffiiaannee, in the paper suggested they mention the Encoder-Decoder Bidirectional RNN, however the implementation presented in the tutorial follows the description of Luong's et al paper. Particularly, they use the concat strategy for score function in the attention mechanism and the architecture uses two RNNs in contrast with the suggested bidirectional RNN on the other's paper.

The main suggestion would be to correct the references in the tutorial.

[bjhargrave]

/assigntome