Failed to compile from sources

[morgan-bc]

Enviroment:

• System: Ubuntun-20.04
• gcc: 9.3.0
• python: 3.8
• python: 1.9.1+cpu
• ipex: master

I followed the README instructions to compile the ipex sources but failed. I also tried to build ipex with torch-1.10+cpu, but failed too. The error message is as follows:

/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/cpu/BatchNorm.cpp: In function ‘at::Tensor torch_ipex::autocast::frozen_batch_norm(const at::Tensor&, const at::Tensor&, const at::Tensor&, const at::Tensor&, const at::Tensor&)’:
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/cpu/BatchNorm.cpp:326:66: error: ‘AutocastCPU’ is not a member of ‘c10::DispatchKey’; did you mean ‘AutocastCUDA’?
  326 |   c10::impl::ExcludeDispatchKeyGuard no_autocastCPU(DispatchKey::AutocastCPU);
      |                                                                  ^~~~~~~~~~~
      |                                                                  AutocastCUDA
In file included from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/jit/runtime/operator.h:13,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/jit/ir/ir.h:7,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/function_impl.h:4,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/method.h:5,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/object.h:6,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/jit/frontend/tracer.h:9,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/autograd/generated/variable_factories.h:12,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:7,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /root/anaconda3/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from /root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/cpu/BatchNorm.cpp:1:
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/cpu/BatchNorm.cpp: At global scope:
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/cpu/BatchNorm.cpp:336:32: error: ‘AutocastCPU’ is not a member of ‘c10::DispatchKey’; did you mean ‘AutocastCUDA’?

For pytorch-1.10, the error message is

root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp: In function ‘void torch_ipex::autocast::TORCH_LIBRARY_IMPL_init_aten_AutocastCPU_84(torch::Library&)’:
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp:164:72: error: could not convert template argument ‘& at::linalg_matrix_rank’ from ‘<unresolved overloaded function type>’ to ‘at::Tensor (*)(const at::Tensor&, double, bool)’
  164 |         &CPU_WrapFunction<DtypeCastPolicy::CAST_POLICY, SIG, SIG, &FUNC>:: \
      |                                                                        ^
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp:445:1: note: in expansion of macro ‘MAKE_REGISTER_FUNC’
  445 | MAKE_REGISTER_FUNC(
      | ^~~~~~~~~~~~~~~~~~
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp:165:13: error: ‘<expression error>::type’ has not been declared
  165 |             type::call);                                                   \
      |             ^~~~
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp:165:13: note: in definition of macro ‘MAKE_REGISTER_FUNC’
  165 |             type::call);                                                   \
      |             ^~~~
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp: At global scope:
/root/downloads/intel-extension-for-pytorch/torch_ipex/csrc/autocast_mode.cpp:168:15: error: template-id ‘get_op_name<at::Tensor(const at::Tensor&, double, bool), at::linalg_matrix_rank>’ for ‘std::string torch_ipex::autocast::get_op_name()’ does not match any template declaration

[EikanWang]

I'm trying to fix it and will ping you later.

[EikanWang]

Hi @morgan-bc , I created a branch for IPEX 1.10 release(release/1.10), it is on top of PyTorch 1.10. So you can install PyTorch 1.10 and then build IPEX 1.10 from the source to fix the error.
Back to the error, the root cause is PyTorch master changed some operators' signatures.

[wduo]

Hi @morgan-bc , I created a branch for IPEX 1.10 release(release/1.10), it is on top of PyTorch 1.10. So you can install PyTorch 1.10 and then build IPEX 1.10 from the source to fix the error. Back to the error, the root cause is PyTorch master changed some operators' signatures.

I use 'IPEX 1.10 release(release/1.10)' and fix the error 'Failed to compile from sources' that use master branch;
then when training, it has error 'RuntimeError: all elements of input should be between 0 and 1' in #26 .

in #26 , EikanWang merged 1 commit into intel:master from JianpingChen066:FixBF16MoveKerBug , so i need to use master branch?

how should i do?

[wduo]

@EikanWang Looking forward to your reply.

[EikanWang]

@wduo , thanks for your report. May I know how to reproduce your issue? According to your description, it should be a regression because the FIXed PR has not been merged to the master and release/1.10 branch. May I know if you apply the PR #29, has the issue gone?

[morgan-bc]

@EikanWang Thanks. I compiled the release/1,10 successfully with Pytorch 1.10, but the ipex cannot work. The ipex crashed with Segmentation fault. The example codes are as follow

import torch
import numpy as np
from tqdm import tqdm
from torchvision.models import resnet18
from torchvision.datasets import CIFAR10
from torchvision import transforms as T
import intel_extension_for_pytorch as ipex

data_dir ='data'
trans = T.Compose([T.ToTensor(), T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])   

model = resnet18(num_classes=10)

train_data = CIFAR10(data_dir, train=True, download=True, transform=trans)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=320, shuffle=True)

optimizer = torch.optim.SGD(model.parameters(), 0.001)
loss_fn = torch.nn.CrossEntropyLoss()

model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.float32)

steps_per_epoch = len(train_loader)

for epoch in  range(10):
    print('epoch:', epoch)
    pbar = tqdm(range(steps_per_epoch), ncols=80)
    for x, y in train_loader:
        #x = x.to(device)
        #y = y.to(device)
        optimizer.zero_grad()
        p = model(x)
        loss = loss_fn(p, y)
        loss.backward()
        optimizer.step()
        pbar.set_postfix_str('loss: {:.4f}'.format(loss.item()))
        pbar.update()
    pbar.close()
       

[wduo]

@EikanWang 您好！

1. 我源码编译了pytorch的release/1.10分支的代码；
2. 然后开始源码编译ipex的master分支代码，报错如 issue Failed to compile from sources  #192 ：

Failed to compile from sources

我按照您的这个回复https://github.com/intel/intel-extension-for-pytorch/issues/192#issuecomment-963881566修复了issue #192，成功的编译了ipex的release/1.10分支的代码。

3. 在训练时，报错如 issue Loss function RuntimeError when use BF16 #26 ：

Loss function RuntimeError when use BF16

从 #26中，我看到了 pr #29，看到这个pr被合并到ipex master中了。

上述是我的流程，您看是否可以根据我的描述重现问题呢？
请问我应该如何将该pr #29应用到ipex release/1.10分支中呢？

[wduo]

@EikanWang 我看了我的模型输出结果：pred =model(data) ，发现tensor中所有值都为nan ，类型为 (dtype=torch.bfloat16, grad_fn=)或(dtype=torch.bfloat16, grad_fn=)

[EikanWang]

@wduo , 如果，使用BF16，会碰到Loss function RuntimeError，原来的#26，fix了这个问题，对吧？

[wduo]

@wduo , 如果，使用BF16，会碰到Loss function RuntimeError，原来的#26，fix了这个问题，对吧？

没有fix, 还是报RuntimeError: all elements of input should be between 0 and 1

[wduo]

@EikanWang 我是按照这个教程运行程序的，报#26：RuntimeError: all elements of input should be between 0 and 1

import torch
import torch.nn as nn
# Import intel_extension_for_pytorch
import intel_extension_for_pytorch as ipex

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = nn.Linear(4, 5)

    def forward(self, input):
        return self.linear(input)

model = Model()
model.set_state_dict(torch.load(PATH))
optimizer.set_state_dict(torch.load(PATH))

# Invoke optimize function against the model object and optimizer object with data type set to torch.bfloat16
model, optimizer = ipex.optimize(model, optimizer, dtype=torch.bfloat16)

for images, label in train_loader():
    with torch.cpu.amp.autocast():
        # Setting memory_format to torch.channels_last could improve performance with 4D input data. This is optional.
        images = images.to(memory_format=torch.channels_last)
        loss = criterion(model(images), label)
    loss.backward()
    optimizer.step()
torch.save(model.state_dict(), PATH)
torch.save(optimizer.state_dict(), PATH)

[EikanWang]

Got it. I will investigate it and go back here.

[EikanWang]

@CaoE , could you help to reproduce this issue?

[wduo]

@EikanWang @CaoE 请问您查出来问题所在了吗？