"Could not copy" (Operation not permitted) when compiling in a docker container

[mattgodbolt]

I'm the maintainer of Compiler Explorer, and recently (in the last 2-3 days) the nightly build of Rust has started reporting an error when I try and compile using it from within a Docker container:

root@c45dac5bb5c7:/tmp# cat > test.rs
pub fn square(num: i32) -> i32 {
    num * num
}
root@c45dac5bb5c7:/tmp# strace -f -o/tmp/moo /opt/compiler-explorer/rust-nightly/bin/rustc -C debuginfo=1 -o /tmp/moose.s --emit asm test.rs --crate-type rlib                                                     
error: could not copy "/tmp/moose.test0.rcgu.s" to "/tmp/moose.s": Operation not permitted (os error 1)

error: aborting due to previous error

The same compile works just fine outside of a container. The strace call yields the problem call:

4809  stat("/tmp/moose.test0.rcgu.s", {st_mode=S_IFREG|0644, st_size=3360, ...}) = 0
4809  open("/tmp/moose.test0.rcgu.s", O_RDONLY|O_CLOEXEC) = 5
4809  open("/tmp/moose.s", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 6
4809  fstat(5, {st_mode=S_IFREG|0644, st_size=3360, ...}) = 0
4809  syscall_18446744073709551615(0x5, 0, 0x6, 0, 0xd20, 0x7f3200000000) = -1 (errno 1)
4809  close(6)                          = 0
4809  close(5)                          = 0
4809  write(2, "\33[0m\33[1m\33[38;5;9merror\33[0m\33[0m\33["..., 137) = 137
4809  write(2, "\n", 1)                 = 1
4809  unlink("/tmp/moose.test0.rcgu.s") = 0

It seems whatever "syscall_18446744073709551615" is, it's getting EPERM.

Cross checking from outside a docker container:

6206  open("/tmp/moose.s", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 6
6206  fstat(5, {st_mode=S_IFREG|0664, st_size=3360, ...}) = 0
6206  syscall_326(0x5, 0, 0x6, 0, 0xd20, 0x7f3f00000000) = -1 (errno 38)
6206  read(5, "\t.text\n\t.file\t\"test0-8787f43e282"..., 8192) = 3360
6206  write(6, "\t.text\n\t.file\t\"test0-8787f43e282"..., 3360) = 3360
6206  read(5, "", 8192)                 = 0
6206  chmod("/tmp/moose.s", 0100664)    = 0
6206  close(6)                          = 0
6206  close(5)                          = 0
6206  unlink("/tmp/moose.test0.rcgu.s") = 0

in this case things succeed (although the syscall_326, whatever that is still fails, but with ENOSYS).

Can anyone shed any light on this as this is outside of my experience! As I say, it only happens on the nightly build, and it has started happening in the last day or two (e.g. sometime around 29th May 2018).

I wonder if some new (unsupported on Ubuntu 16) libc function is being called, and in the docker environment EPERM is returned instead of ENOSYS, which confuses some bailout code in rustc?

Thanks in advance, Matt

(See also compiler-explorer/compiler-explorer#942 )

[mattgodbolt]

syscall_326 is dup3 - from the man page: dup3() was added to Linux in version 2.6.27; glibc support is available starting with version 2.9.

Seems the dup3 call isn't supported in the environment, and then something inside docker is getting confused (it's using syscall_INT_MAX`). I'm not sure what the right solution is here.

[alexcrichton]

cc @nicokoch and #50772

[nicokoch]

I just took a brief look at this.

@mattgodbolt The compiler recently started to support copy_file_range instead of the more traditional read/write to copy files.
What is actually happening here is that the system call is disallowed in your docker configuration (seccomp). The system will return EPERM in this case. The system call started being allowed by default in docker 1.11.0 though. So the "easy" fix on your side is allowing the system call in your seccomp configuration.

@alexcrichton Technically, this bug's cause is misconfiguration of docker. However, to avoid issues like this in the future, it may make sense to add "EPERM" to the fallback error cases here https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/fs.rs#L905

What do you think?

[nicokoch]

As I think more about it, we should definitely add E_PERM to the fallback error cases.

If we have a "real" permission denied (in the sense that file permissions are not sufficient), we would get the error earlier here https://github.com/rust-lang/rust/blob/master/src/libstd/sys/unix/fs.rs#L870 . So there's actually no downside to adding E_PERM to the fallback error cases.

I'll file a PR.

[nicokoch]

One more note: EPERM isn't even a valid error code for copy_file_range according to http://man7.org/linux/man-pages/man2/copy_file_range.2.html .
That's why it wasn't accounted for in the first place.

More documentation on docker + seccomp can be found here: https://docs.docker.com/engine/security/seccomp/

[mattgodbolt]

Thanks @nicokoch - I'll look to update docker on my side anyway. I appreciate the discussion here, and that EPERM is a very surprising result from copy_file_range!

[mattgodbolt]

I've confirmed a newer docker fixes the issues too. Thanks so much for the quick responses here!