Transfer learning tutorial -- momentum handling

[lizhitwo]

In the tutorial, a new SGD optimizer is generated each epoch, which means that the momentum from the last epoch is discarded. As far as I can tell, this is unusual behavior in deep network training. The same optimizer should be used, but with learning rate modified.

[chsasank]

There's a discussion about this on pytorch forums here

@apaszke suggests

If you want to change the LR we recommend reconstructing the optimizer with new parameters.

Your issue has also been raised in that thread, although there has been no conclusive answer yet. May be @apaszke or @soumith will help us with this?

[lizhitwo]

Thanks! It seems the best way for reproducing the usual behavior right now is to manually adjust the learning rate for each group.

[chsasank]

Hi @lizhitwo,

I've created a PR #62 for this. Can you please have a look before I merge and build?

[lizhitwo]

This looks right. Thanks!

[chsasank]

Thanks for raising the issue.