Pedagogically confusing image size in "NEURAL NETWORKS"

[WillRNaylor]

At the start of the tutorial:
https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html

You have 5x5 convolutions and in both of the Conv layers and then you add the next linear layer explicitly writing that it has size (16 * 5 * 5, 120). Clearly the 16 comes from the number of filters in the prev. layer, and it seems at first obvious that the hard coded 5 * 5 might come from the filter size, but it does not - it should come from the dimension of the input of the image (which is 32 x 32). You could make a note of this, or simply change the size of the filters to 3 x 3 or 7 x 7 or something.

Of course the code currently runs fine, and I see you added the nice pedagogical function def num_flat_features(self, x), which was what helped me realise something was amiss in the code.

Below is the init of the Net() class which confused me for reference.

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

There is also an issue here, which is maybe related.

#498

Apologies if this is well written, and thanks to all who made the great PyTorch intro pages.

[soumith]

this seems very reasonable to attempt to fix, so that future readers wont get affected by it.

Would you be interested in sending a PR to fix the sample code, by editing this file: https://github.com/pytorch/tutorials/blob/master/beginner_source/blitz/neural_networks_tutorial.py ?

[WillRNaylor]

So, I just went ahead and made the simplest pull request:

#515

Note that I went for 3x3 filters over 7x7 as the input images are already small (32x32), so with 7x7 we would end up with a 4x4 encoded "image", after the conv. layers. It's 6x6 as you see in the code with the 3x3 filters.

Thanks, and I will take no offence if you think of a better solution!

[soumith]

closed via #515

[saper0]

I stumbled accross this issue doing the PyTorch tutorial. As of now, the image showing the (LeNet) architecture of the NN and the implementation do not match even though in the text it is claimed that the shown architecture is implemented by the given code. This confused me quite a bit while working through the tutorial and let me question at first if I have misunderstood certain aspects about CNNs as I didn't expect such an error.

Is it possible to note that the given implementation does not match the shown architecture and change this wrong claim in the tutorial description? Or is it possible to change the filter sizes back to 5x5 and include a note, as first proposed by @WillRNaylor , that the 16 * 5 * 5 linear layer input size results due to the max pooling layer and has nothing to do with the filter size beeing 5x5.

In the current implemention, as also noted by @WillRNaylor in #515 , the max pooling layer with a 2x2 filter is applied on a 13x13 image leading to cutting away/not using the data in the last row and column which seems to me to be a bad NN design decision. Therefore rechanging the filter size back to 5x5 and leaving a clarifying note seems to me to be the better and more pedagogical solution.

[saper0]

I want to extend my above comment as I have worked through the follow up tutorial Training a Classifier and found another inconsistency in the current tutorial caused by #515 . In Training a Classifier it is stated to copy the CNN from the previous Neural Networks tutorial and to extend it to three input channels instead of one. But the implementation in Training a Classifier still uses 5x5 filters.

[WillRNaylor]

Sorry for the late reply. But I'm not sure I fully follow this. The image on the page I liked to above clearly shows a different architecture to what's in the code (in the code we have two conv. layers, and there is certainly no 28x28 filters. I also don't see where it claims that the implementation matches the image, so I suspect we are talking about different pages.

As for cutting off the final layer of pixels when going through the final dense layers (with a 2x2 max_pool on a 13x13 image). Although it's not "optimal", it's not pedagogically that bad, as it's a perfectly legitimate thing to do, and something that may not occur to beginners (as on a real sized image a single layer of pixels may not be worth the change in filter size.

As for the second comment, this is a great pick up, and you are quite right. We state that we are reusing the code from before, but we aren't. I personally don't really mind what you do, either going back to 5x5 and adding a note - or adding a note to the "Training a Classifier", or changing that code.

[matsui528]

Hi @WillRNaylor
In the figure, the shape of C1: feature maps is defined as 6@28x28. This is the result of 5x5 conv over the 32x32 image. Now, the 5x5 conv is replaced by 3x3 conv. So the description in the figure should be updated to 6@30x30?

I think this is the very beginning tutorial, so it would be helpful for beginners to make the description very precise :)

Related: 6308024#commitcomment-34816516