nvfuser tutorial needs to be updated for 1.13

[zou3519]

From offline discussion with @ptrblck and @Chillee. I'm putting up this issue so that the release team is aware of this item.

[IvanYashchuk]

Is it this issue pytorch/pytorch#86020 ?

@Chillee, are you going to revert cache removal? pytorch/pytorch#86020 (comment) facebookresearch/xformers#468 (comment)

[svekars]

Should this be moved to the tutorials repo?

[zou3519]

Should this be moved to the tutorials repo?

Fix is either to revert the cache removal (in PyTorch) or update the tutorial, so I'd keep this issue here until we align on how to proceed

[Chillee]

@IvanYashchuk I discussed with the xformers folks, and given the recent merge of Dynamo into core and our general discussion, they'd prefer to just use dynamo directly and so didn't need the cache reverted.

From my discussion with NVFuser folk, it also seemed fine to not readd it. So, no, don't plan on reverting that PR.

[IvanYashchuk]

I know that all the code in the functorch module is experimental and subject to rapid changes. It's still weird to potentially break already published tutorials that in part promote functorch right before the branch cut.

I have verified that tutorial runs fine (including functorch.compile.memory_efficient_fusion call) with 1.14.0.dev20221020+cu116 on Colab, so I assume it would continue to be executable with 1.13.0 as well.

[zou3519]

I checked -- I executed the tutorial against the 1.13.0 RC. It runs fine. The caveat is that a user using memory_efficient_fusion directly may run into unsafe behavior due to the removal of the cache, but, that is why we're moving toward torchdynamo as the frontend for the compilation stack.

I know that all the code in the functorch module is experimental and subject to rapid changes. It's still weird to potentially break already published tutorials that in part promote functorch right before the branch cut.

I agree it's not good to break tutorials close to release time. The tutorial is not technically broken, as I mentioned above. Though I do want to clarify that we do not offer stability guarantees on functorch.compile, but we do on all the other functorch APIs (those are released as "beta").

[zou3519]

For action items -- @IvanYashchuk do you think we should update the nvfuser tutorial to say anything?

[IvanYashchuk]

I think - yes - because right before the functorch section dynamic shapes are discussed. And as it was pointed out here pytorch/pytorch#86020 (comment) memory_efficient_fusion should be re-constructed manually for every new input shape.

There's a note in the tutorial:

FuncTorch’s memory efficient pass is still actively in development and future versions are expected to achieve performance closer to that of TorchScript with the composite definition.

So maybe that's where one sentence about stability guarantees would just partially guard the PyTorch repo against unnecessary bug reports in the future?

Thank you for verifying it with 1.13.0 RC!

[zou3519]

I can take a stab, would just need someone from the Nvidia side to review. Migrating this issue to the tutorials repo because there is only a tutorials question