IPEX launch script performance in docker

[Peach-He]

Hi IPEX team,

Thanks for excellent project, I'm having weird performance issue when using IPEX launch scripts in docker, that disabling thread affinity delivers 1.28x training speedup which is unexpected. Do you know the possible root cause?

Bellow is detailed information:

• system config: 4 nodes ICX, pytorch 1.10, ipex 1.10
• docker image based on intel oneapi-aikit
• training scripts: python -m intel_extension_for_pytorch.cpu.launch --distributed --nproc_per_node=2 --nnodes=4 --hostfile hosts train.py

How I disable thread affinity:

# modify launch.py, comment out following code
self.set_env("KMP_AFFINITY", "granularity=fine,compact,1,0")
self.set_env("I_MPI_PIN_DOMAIN", mpi_pin_domain)
self.set_env("CCL_WORKER_AFFINITY", ccl_affinity)

Looking forward to your reply, thanks.

[jgong5]

@Peach-He Are you able to repro the problem with the latest IPEX (1.12)? cc @jingxu10

[Peach-He]

@jgong5 I only use IPEX launch script to launch my training script, no other IPEX optimization was added yet. I think there should be difference using 1.10 launch script or 1.12, right?

[jingxu10]

Could you share a test case for us to reproduce this issue?
Distributed training part of 1.10 and 1.12 should be the same.
Did you use torch-ccl with this script? If you did, which version?

[Peach-He]

Sorry for late reply.
Take IPEX distributed training example here, comment out model, optimizer = ipex.optimize(model, optimizer=optimizer), I can also see about 3% perforamnce regression w/o thread affinity in docker (the performance regression is much smaller than my own workload, but still no thread affinity runs faster than thread affinity).

Did you use torch-ccl with this script? If you did, which version?

yes, torch ccl version is 1.10.0, ipex version is 1.10.100