Add forward AD tutorial

[soulitzer]

No description provided.

[netlify]

✔️ Deploy Preview for pytorch-tutorials-preview ready!

🔨 Explore the source changes: 111f2d5

🔍 Inspect the deploy log: https://app.netlify.com/sites/pytorch-tutorials-preview/deploys/61a956aa5fb4d1000846cd65

😎 Browse the preview: https://deploy-preview-1746--pytorch-tutorials-preview.netlify.app

[soulitzer]

Gradcheck should not require inputs to require grad if we are not checking backward AD. See: pytorch/pytorch#69004

[albanD]

Not sure this is true.
We do use requires_grad to know which input/outputs are differentiable.

Maybe we should not, but today we do require this, and just removing the check doesn't sound like the right solution.

[soulitzer]

You're correct about the inputs, but looks like gradcheck currently doesn't actually need the outputs to have requires grad:

import torch
import torch.autograd.forward_ad as fwAD

def detach_backward(x):
  _, tangent = fwAD.unpack_dual(x)
  if not tangent:   # non-dual-level friendly path
    return x.detach()
  return fwAD.make_dual(x.detach(), tangent)

class Func(torch.autograd.Function):
  @staticmethod
  def forward(ctx, x):
    return x * 2

  @staticmethod
  def jvp(ctx, gI):
    return gI * 3  # Wrong, lets see if gradcheck catches this

t = torch.tensor(1., requires_grad=True, dtype=torch.double)

def fn(x):
  out1 = detach_backward(Func.apply(x))
  out2 = x.clone()  # only necessary to bypass the no differnetiable outputs check
  return out1, out2

with fwAD.dual_level():
  dual = fwAD.make_dual(t, torch.rand_like(t))
  out = fn(dual)
  assert not out[0].requires_grad

torch.autograd.gradcheck(fn, (t,), check_forward_ad=True,
  check_backward_ad=False, check_batched_grad=False, check_undefined_grad=False)

I don't know whether the fact non-require-grad outputs are checked is useful though. At least for custom functions, it looks like the output still requires_grad even if only jvp is implemented, i..e, does a function that behaves like detach_backward actually exist in nature?

[albanD]

does a function that behaves like detach_backward actually exist in nature?

Ignoring things like detach, I think the answer is no. If a function is differentiable, it will be for both forward and backward.

[albanD]

Note that this needs an API that is only in master so the build will fail.
You can rename the file so that it doesn't end with "_tutorial" such that it won't run. We can do a follow up PR for 1.11 that moves it back to make it run again.

[albanD]

sgtm